Meaningful Metrics for Evaluating Eventual Consistency

Optimistic replication is a fundamental technique for supporting collaborative work practices in mobile environments. However, eventual consistency, in contrast to immediate strong consistency in pessimistic replication, is much harder to evaluate. This paper analyzes different metrics for measuring the effectiveness of eventually consistent systems. Using results from a simulated environment of relevant optimistic replication protocols, we show that each metric hides previously undocumented side effects. These add considerable imprecision to any evaluation that exclusively relies on a single metric. Hence, we advocate a combined methodology comprising three complementary metrics: commit ratio, average agreement delay and average commit delay.